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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

Appellant: Bradley C. Love 
Serial No.: 09/686,112 
Filed: 10/10/2000 



On Appeal to the 
Board of Appeals 



For: "Method and Apparatus for Incorporating 
Decision Making Into Classifiers" 

Mail Stop Appeal Briefs - Patents 
Hon. Commissioner for Patents 
P.O. Box 1450 
Alexandria, VA 22313-1450 



Examiner: Hirl, Joseph P 
Group Art Unit: 2121 
Our Ref: HRL030 



BRIEF ON APPEAL 



Sir: 



This is an appeal from the Final Rejection, dated October 21, 2003, for the above- 
identified patent application. 

REAL PARTY IN INTEREST 



27 The present application has been assigned to HRL Laboratories, LLC of Malibu, 

28 CA. 



RELATED APPEALS AND INTERFERENCES 

There are no related appeals or interferences to this application. 

STATUS OF CLAIMS 

Claims 1, 6, 7, 10, 16, 17, 20, and 26 are rejected. 
Claims 2-5, 8, 9, 1 1-15, 18, 19, and 21-25 are objected to. 



36 Claims 27 and 28 are not under any objections or rejections. 
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Claims 1 5 6, 7, 10, 16, 17, 20, 26 are the subject of this appeal. A copy of all 



2 claims of the application is contained in the attached Appendix A. 



3 



4 



STATUS OF AMENDMENTS 



5 



No Amendment after Final Rejection has been entered. 



6 



7 



SUMMARY OF INVENTION 



8 



A method and apparatus for incorporating decision making into classifiers to 



9 provide efficient test recommendations are presented and claimed. The apparatus 

10 comprises an explicit system and a classifier, each configured to receive a system state 

1 1 dataset, with the explicit system connected with the classifier, and operative to iteratively 

12 perform a combinatory search procedure based on the system state dataset to develop a 

13 next test recommendation for the classifier, whereby the classifier performs the next test 

14 to generate an objective weighted score. The system state dataset is typically in the form 

15 of a vector representing the various features that describe the System State. The 

16 apparatus further includes a profit module connected with the classifier and with the 

17 explicit system to receive the objective weighted score from the classifier, to add 

1 8 subjective value to the objective weighted score to determine a profit for the test, and to 

19 provide the profit to the explicit system to enable the explicit system to assess the value 

20 of its next test recommendation, and, iteratively, to generate a best test recommendation 

21 based on the maximization of the profit. The apparatus further includes an implicit 

22 system configured to receive a system state dataset, and connected with the explicit 

23 system to receive the best test recommendation for each system state dataset, and to act as 
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1 a function estimator to learn to associate best test recommendations with the system state 

2 dataset in order to mimic the explicit system, thereby to enable rapid decision making in 

3 situations that are either urgent or well-known. 
4 

5 Although the implicit system may be any probability-based learning system, it is 

6 preferably a neural network. The combinatory search procedure performed by the 

7 explicit system is preferably simulated annealing. 
8 

9 After the implicit system has been sufficiently trained to mimic the explicit system, the 

10 explicit system and the profit module may be separated from the implicit system in order 

1 1 to provide a smaller, lower cost system that approximates the performance of the overall 

12 system. This application is especially useful when the system is dealing with a fixed 

13 environment, i.e. there are few changes that would be outside the range in which the 

14 implicit system was trained. 
15 

16 ISSUES 

17 Issue 1 - Whether Claims 7, 17, and 26 lack compliance with 35 USC 1 12, fourth 

18 paragraph, in that these claims fail to additionally limit the subject matter of the related 

1 9 independent claim. 

20 Issue 2 - Whether Claims 1 and 20 and Claims 6 and 16 are indefinite under 35 

21 USC 1 12, second paragraph, for failing to particularly point out and distinctly claim the 

22 subject matter which Appellant regards as the invention. In particular, whether the terms 
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1 "mimic" as used in Claims 1 and 20, and "sufficiently mimics" as used in Claims 6 and 

2 16, are relative terms that render the claims indefinite. 

3 Issue 3 - Whether Claim 10 is indefinite under 35 USC 112, second paragraph, for 

4 failing to particularly point out and distinctly claim the subject matter which Appellant 

5 regards as the invention. In particular, whether the preposition "on" used in Claim 10 is 

6 confusing. 

7 GROUPING OF CLAIMS 

8 The grounds of rejections and objections which appellant contests herein apply to 

9 more than one claim, such additional claims, to the extent separately identified and 

10 argued below, do not stand or fall together. 

11 THE ARGUMENT 

12 Issue 1 - Whether Claims 7, 17, and 26 lack compliance with 35 USC 112, 

1 3 fourth paragraph, in that these claims fail to additionally limit the subject matter of the 

1 4 related independent claim. 
15 

16 Claims 7 recites: 

17 An apparatus for incorporating decision making into classifiers to provide 

1 8 efficient test recommendations as set forth in claim 1, wherein the System State is a 
• 19 vector. 

20 

21 Claim 17 recites: 
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1 A computerized method for enhancing decision making into classifiers to provide 

2 efficient test recommendations as set forth in claim 10, wherein the System State is a 

3 vector. 
4 

5 Claim 26 recites: 

6 A computerized method for enhancing decision making in classifiers to provide 

7 efficient test recommendations as set forth in claim 20, wherein the System State is a 

8 vector. 
9 

10 In the first Office Action dated May 7, 2003, the Examiner objected to Claims 7, 

11 17, 26 under 37 CFR 1 .75(c) for lack of compliance with 35 USC 1 12, fourth paragraph, 

12 in that these claims fail to additionally limit the subject matter of the related independent 

13 claim. The Examiner stated that "by definition, the system state is always a vector. " 
14 

15 In the Response to the first Office Action dated May 7, 2003, the Appellant stated 

16 that "[t]he first partial paragraph on page 19 of the present application states, "Note 

1 7 that representing the problem as a vector is appropriate only when the test costs are not 

1 8 affected by which other tests have already been performed. " Thus, it is not true that the 

19 system state is always a vector as asserted by the Examiner. Therefore, the Applicant 

20 submits that Claims 7, 17, and 26 are in compliance with 35 USC 112, fourth paragraph, 

21 as they additionally limit the subject matter of the related independent claim. If the 

22 Examiner still maintains his objection to these claims, the Applicant respectfully requests 
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1 that the Examiner point out to the Applicant his basis for his interpretation that, by 

2 definition, the system state is always a vector. " 
3 

4 In the Final Office Action of October 21 , 2003, the Examiner finally objected to 

5 claims 7, 17, and 26 under 37 C.F.R. 1.75(c) for lack of compliance with 35 U.S.C. 1 12, 

6 fourth paragraph. The Examiner stated that " these claims fail to additionally limit the 
1 subject matter of the related independent claim in that by definition the system state is 

8 always a vector. " In response to the Appellant's arguments and a request that the 

9 Examiner elaborate and clarify his basis for the objection in the First Office Action dated 

10 May 7, 2003, the Examiner further stated that "ft]o one of ordinary skill in the art, the 

1 1 system state is defined by n coordinates in Euclidean n-space. A vector is a well-known 

1 2 mathematical concept defined by n coordinates in Euclidean n-space. The well accepted 

13 methodology of vectors is commonly used to refer to the n coordinates of the system state 

14 in Euclidean n-space. " 
15 

16 The following are direct quotes regarding the term "vector" used throughout the 

17 disclosure: 
18 

19 Page 17, line 22 to page 18, line 5 states: 

20 In the context of choosing the best subset of tests, a local search algorithm starts 

21 out at some state, typically in the form of a vector with an entry for each test indicating 

22 whether or not the given test is to be performed. The local search algorithm changes 
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1 entries in the vector to indicate whether the test will be run, and accepts a change when 

2 the expected profit rises. 
3 

4 As the above quoted section clearly indicates, the system state is typically in the 

5 form of a vector, and therefore is not limited to a vector form of representation. 
6 

7 Page 19, lines 3 to 9 state: 

8 "Note that representing the problem as a vector is appropriate only when the test 

9 costs are not affected by which other tests have already been performed. FIG. 3(b) 

1 0 represents a problem in which test costs are affected by which other tests have already 

1 1 been performed. In such cases, optimization occurs over a matrix. " 
12 

1 3 The Appellant respectfully disagrees with the Examiner's assertion that the 

14 system state is always a vector, and particularly the belief that the System State is defined 

15 by n coordinates in Euclidean n-space. The System State of the present application 

1 6 cannot simply be described as a vector, with values that are known and unknown. The 

1 7 System State is in actuality much more broad in its scope, and can include multiple test 

1 8 dependencies which exist outside of the simple bounds of a vector. As stated on page 1 9, 

19 lines 3 to 9, (i [n]ote that representing the problem as a vector is appropriate only when 

20 the test costs are not affected by which other tests have already been performed. " This 

21 describes the situation in which a test dependency would exist that would bring the 

22 System State outside of the traditional definition of a vector. Essentially, when the 

23 system is searching for the next test to evaluate, it must navigate a matrix structure for 
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1 situations in which there are test dependencies. The state of the search process is part of 

2 the System State, and therefore cannot be limited to the definition of a simple vector, 

3 which is limited to known and unknown values. Appendix B includes a declaration by 

4 the inventor, one of ordinary skill in the art, as to the interpretation of the System State as 

5 more than a simple vector. Furthermore, the complete System State of the model can 

6 additionally encompass the specification of the classifier, the implicit system, and the 

7 explicit system. Thus, the System State is not defined by n coordinates in Euclidean n- 

8 space, but is subject to a much broader interpretation. 
9 

10 Based on the prior comments regarding the definition and scope of the System 

1 1 State, Appellant believes that Claims 7, 17, and 26 are in compliance with 35 USC 1 12, 

12 fourth paragraph, as they additionally limit the subject matter of the related independent 

13 claim. 
14 

1 5 Issue 2 - Whether Claims 1 and 20 and Claims 6 and 16 are indefinite under 35 

1 6 USC 112, second paragraph, for failing to particularly point out and distinctly claim 

1 7 the subject matter which Appellant regards as the invention. In particular, whether the 

1 8 terms "mimic" as used in Claims 1 and 20, and "sufficiently mimics" as used in 

1 9 Claims 6 and 16, are relative terms that render the claims indefinite. 
20 

21 Claim 1 recites, inter alia: 

22 "...an implicit system configured to receive a system state dataset, and connected 

23 with the explicit system to receive the best test recommendation for each system state 
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1 dataset, and to act as a function estimator to learn to associate best test 

2 recommendations with the system state dataset in order to mimic the explicit system, 

3 thereby to enable rapid decision making in situations that are either urgent or well- 

4 known. " 

5 

6 Claim 20 recites, inter alia: 

7 "...providing an implicit system configured to receive a system state dataset, and 

8 connected with the explicit system to receive the best test recommendation for each 

9 system state dataset, and to act as a function estimator to learn to associate best test 

10 recommendations with the system state dataset in order to mimic the explicit system, 

1 1 thereby to enable rapid decision making in situations that are either urgent or well- 

12 known." 

13 

14 In the first Office Action dated May 7, 2003, the Examiner rejected Claims 1 and 

15 20 under 35 USC 112, second paragraph, as being indefinite for failing to particularly 

16 point out and distinctly claim the subject matter which Appellant regards as the invention. 

17 The Examiner stated that "At Claim 6, line 4 the term "mimic " is used which is a relative 

1 8 term and renders the [sic J claim indefinite. " 

19 

20 Appellant believes the Examiner has made an error in arguing claim 6 under the 

21 35 USC 1 12, second paragraph, rejection of claims 1 and 20. The term "mimic" appears 

22 in paragraph "c" of both claims 1 and 20. In Response to the First Office Action, the 
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1 Appellant respectfully disagreed with the Examiner, and stated that "ft] he present 

2 application, page 16, lines 15-1 7 states "Although the explicit system 202 uses a 

3 combinatorial process to determine the best test or tests to run, over time the implicit 

4 system 206 learns to mimic the performance of the explicit system 202. " The Applicant is 

5 unclear why the Examiner has asserted that the term "mimic " is relative. The Applicant 

6 respectfully requests that if the Examiner continues this rejection of Claims 1 and 20 
1 based on the term (t mimic " that the Examiner explain why he believes the term is 

8 relative. Because the term "mimic " is used in a definable context in the present 

9 application, the Applicant submits that one skilled in the art would understand what the 

10 term "mimic" means and thus, the term "mimic" is not indefinite. Therefore, the 

1 1 Applicant respectfully requests that the Examiner withdraw this rejection of Claims 1 and 

12 20 " 
13 

14 In the Final Office Action, dated October 21, 2003, the Examiner finally rejected 

15 Claims 1 and 20 under 35 USC 1 12, second paragraph, as being indefinite for failing to 

16 particularly point out and distinctly claim the subject matter which the Appellant regards 

1 7 as the invention. The Examiner stated that "...To one of ordinary skill in the art, the term 

1 8 "mimic " introduces the concept of imitation and with the concept of imitation comes the 

1 9 question of "how much "? Since such a term introduces a sliding scale of interpretation, 

20 by consequence the claim is indefinite. " 
21 

22 Claim 6 recites: 
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1 "An apparatus for incorporating decision making into classifiers to provide 

2 efficient test recommendations as set forth in claim 5, wherein the explicit system and the 

3 profit module may be separated from the apparatus after the implicit system sufficiently 

4 mimics the explicit system. " 
5 

6 Claim 16 recites: 

7 "A computerized method for enhancing decision making in a classifier system as 

8 set forth in claim 15, wherein the explicit system and the profit module used may be 

9 separated from the classifier system after the implicit system sufficiently mimics the 
1 0 explicit system. " 

11 

12 In the first Office Action, dated May 7, 2003, the Examiner rejected Claims 6 and 

13 16 under 35 USC 1 12, second paragraph, as being indefinite for failing to particularly 

14 point out and distinctly claim the subject matter which Appellant regards as the invention. 

15 The Examiner stated that "At Claim 6, line 4 the term "sufficiently mimics" is used which 

16 is a relative term and renders the claims indefinite. Claim 16 has a similar reference. " 

17 

1 8 In Response to the First Office Action, the Appellant respectfully disagreed with 

19 the Examiner, and stated that "The Applicant disagrees for the same reasons given above 

20 with reference to claims 1 and 20. Therefore, the Applicant respectfully requests that the 

21 Examiner withdraw this rejection of claims 6 and 16. " 
22 
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1 The following are direct quotes regarding the term "mimic" used throughout the 

2 disclosure: 
3 

4 Page 16, lines 14+: 

5 "Although the explicit system 202 uses a combinatorial process to determine the best test 

6 or tests to run, over time the implicit system 206 learns to mimic the performance of the 

7 explicit system 202. In situations that are familiar or when a very fast approximation to 

8 the performance of the explicit system 202 is required, the implicit system can be used to 

9 provide the final decision. In cases where there are a limited number of possible 

10 situations, the explicit system 202 may be removed after the implicit system 206 has been 

1 1 fully trained. " 
12 

13 Page 20, lines 15+: 

14 "The implicit system compliments the explicit system by providing almost instantaneous 

1 5 test recommendations. The implicit subsystem is a function approximator that, over time, 

1 6 learns to approximate the performance of the explicit system. Thus, the explicit system 

1 7 provides the teaching signal for the implicit system, which will be discussed in greater 

1 8 detail further below. " 
19 

20 Page 26, lines 4+: 

2 1 "In time critical situations, the implicit system can be queried, while the explicit system 

22 can be queried when time is not of the essence. When processor load is low, the explicit 

23 system can provide training data to refine the implicit system. In an alternative 
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1 embodiment, such as a deployed system with limited processing power, the implicit 

2 system can be completely trained by the explicit system while off-line, and the present 

3 invention may be deployed without the explicit system. " 
4 

5 Page 27, lines 1-3: 

6 "After some training, this input/output mapping closely approximates the output of the 

7 explicit system, but requires less computation. " 
8 

9 Appellant asserts that the term "mimic" is not relative to the point that the claims 

10 are rendered indefinite. To one of ordinary skill in the art, the notion of a learning model 

1 1 coming to approximate, estimate, or mimic some underlying function is accepted in 

12 machine learning and statistics (See Appendix B). In the present invention, the function 

13 being approximated is the one defined by the input/output mapping of the explicit system, 

14 as stated in the application on page 20, lines 15-19 (see infra page 12). In each 

15 application of the learning model, there will be different costs associated with gathering 

16 training data, making timely predictions, and making errors. The present invention 

17 provides for these varying costs by adjusting the amount of the explicit system to be 

1 8 mimicked by the implicit system, as stated in the disclosure on page 16, lines 14-20 and 

19 page 26, lines 4-9 (see infra page 12). Thus, what constitutes adequate mimicry cannot 

20 be defined outside of the context of a specific application, since the costs associated with 

21 gathering the data, processing the information, and making accurate predictions will vary 

22 for each application. While mimicking the outputs of the system 60% of the time may be 

23 sufficient in some situations, matching the outputs 99.9% of the time may be inadequate 
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1 in others. Therefore, to be specific as to the amount of mimicry occurring would be 

2 unnecessarily limiting, as the amount of mimicry varies depending on the type and 

3 amount of data and the desire for efficiency based upon the resources available to the 

4 system at any time. 
5 

6 Appellant thus believes that the terms "mimic, 55 as used in Claims 1 and 20, and 

7 "sufficiently mimics", as used in Claims 6 and 16 is not indefinite under 35 USC 1 12, 

8 second paragraph, for the reasons discussed above. 
9 

10 Issue 3 - Whether Claim 10 is indefinite under 35 USC 112, second paragraph, 

1 1 for failing to particularly point out and distinctly claim the subject matter which 

12 Appellant regards as the invention. In particular, whether the preposition "on" used in 

1 3 Claim 10 is confusing. 
14 

15 Claim 10 recites: 

1 6 "A computerized method for enhancing decision making in a classifier system, 

1 7 wherein the classifier system includes an explicit system and a classifier, each configured 

18 to receive a system state dataset, with the explicit system connected with the classifier; a 

1 9 profit module connected with the classifier and with the explicit system; and an implicit 

20 system configured to receive a system state dataset, and connected with the explicit 

21 system, the computerized method comprising the steps of: 

22 a. receiving a system state dataset in the explicit system, the classifier, and the 
2 3 implicit system; 
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1 b. determining in the explicit system, based on the feature set, a recommended 

2 test; 

3 c. performing the recommended test on the classifier; 

4 d. determining, via the profit module, the profit from the test performed on the 

5 classifier; 

6 e. detecting whether the test performed on the classifier maximizes the profit; 

1 f performing the receiving step a through the detecting step e until a test is found 

8 which maximizes the profit; 

9 g. training the implicit system with the system state dataset and the test which 

10 maximizes the profit; and 

1 1 h. repeating steps a through g until a desired level of training of the implicit 

12 system is reached. " 
13 

14 In the First Office action, dated May 7, 2003, the Examiner rejected Claim 10 

15 under 35 USC 1 12, second paragraph, as being indefinite for failing to particularly point 

16 out and distinctly claim the subject matter which Appellant regards as the invention. The 

17 Examiner stated that "fajt Claim 10, lines 10-13, the use of the preposition "on" is 

1 8 confusing [sic] it is really the classifier that is evaluating the test.. .perhaps the word "by " 

19 could be used as a replacement. Without such change, the claim is indefinite. " 
20 

21 In Response to the First Office Action, the Appellant stated that "The last 

22 paragraph on page 15 of the present application states, The explicit system 202 runs 
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1 virtual or hypothetical tests on the classifier 200, which, in essence, acts as a model of 

2 the world in which the system operates, 1 Thus, as disclosed in the present application, 

3 the tests are preformed on the classifier. The Applicant submits that the use of the 

4 preposition "on " is appropriate in this case. Therefore, the Applicant respectfully 

5 requests that the Examiner withdraw this rejection to Claim 10. " 
6 

7 In the Final Office Action, dated October 21, 2003, the Examiner finally rejected 

8 Claim 10 under 35 USC 112, second paragraph, as being indefinite for failing to 

9 particularly point out and distinctly claim the subject matter which Appellant regards as 

10 the invention. The Examiner repeated the reasoning given in First Office Action, and 

1 1 further stated that "...Specification, Fig. 2 applies. Note that the Explicit System is 

12 separate and distinct from that of the Classifier. This means that the Classifier will run 

13 the recommended test. Hence, the word "by" is correct. These comments apply to steps 

14 c, d f and e of claim 10. See also claim 1, step a wherein 'classifier performs. '" 
15 

16 The following are direct quotes regarding the term "on" with respect to the classifier used 

1 7 throughout the disclosure: 
18 

19 Page 15, lines 20+: 

20 "The explicit system 202 runs virtual or hypothetical tests on the classifier 200, which, in 

2 1 essence, acts as a model of the world in which the system operates. " 
22 

23 Page 16, lines 2-9: 
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1 "The profit module 204 tempers the output of the classifier 200 in order to add subjective 

2 cost information relative to the set of tests performed in order to determine the profit 

3 received by performing the set of tests, i.e. output of the profit module 204 reflects the 

4 fact that with each test, there is an associated cost, and that accuracy must be sacrificed 

5 if the cost of the test is simply too high to bear. The output of the profit module 204 is 

6 provided to the explicit system 202 so that the explicit system 202 may use the feedback to 

7 set benchmarks en route to maximizing the profit from the tests. After performing a set of 

8 tests based on the feature set input, the explicit system 202 reaches a final decision of the 

9 best test or tests to run. " 
10 

11 Page 25, lines 5-8: 

12 "The classifier 200 is used essentially as a model of the task environment, or world, with 

13 experiments being run on the classifier to make predictions about what will transpire in 

14 the actual environment. The more veridical the classifier 200 is, the more accurate its 

15 predictions will be. " 
16 

17 Page 26, lines 21+: 

1 8 "In effect, the classifier is used as a model of the task environment and the present 

1 9 invention runs experiments on this model to determine what action to perform next. " 
20 

21 The present application discloses in Claim 10 and throughout the disclosure that 

22 the Explicit System performs the recommended test on the Classifier. Specifically, where 

23 Claim 10 states in steps b and c, "b. determining in the explicit system, based on the 



Page 17 of 20 



US App. No. 09/686,112 



Appeal Brief 



1 feature set, a recommended test; c. performing the recommended test on the classifier;" 

2 the implication of these sections of the claim is that the Explicit System determines a 

3 recommended test and thereafter performs the recommended test on the Classifier. The 

4 steps of the Explicit System performing a test upon the Classifier are further described in 

5 the specification at Page 15, line 20-21, which explains how "[t]he explicit system 202 

6 runs virtual or hypothetical tests on the classifier 200, which, in essence, acts as a model 

7 of the world in which the system operates." While the Examiner cites to Fig. 2 of the 

8 Specification for the assertion that the Explicit System is separate and distinct from the 

9 Classifier, this does not refute the fact that the Explicit System still performs the 

10 recommended test on the Classifier as denoted in Claim 10 and throughout the above- 

1 1 referenced sections of the specification. 

12 In Claim 1, step a, which states "an explicit system and a classifier each 

1 3 configured to receive a system state dataset, with the explicit system connected with the 

14 classifier, and operative to iteratively perform a combinatory search procedure based on 

1 5 the system state dataset to develop a next test recommendation for the classifier, whereby 

16 the classifier performs the next test to generate an objective weighted score;" (italics not 

17 present in original), the Explicit System is described as connected with the Classifier, 

18 where the Explicit System functions to run a test to determine the appropriate test to run 

1 9 on the Classifier, with the Classifier thereafter performing the test part of providing the 

20 output of the test to generate the objective weighted score. While admittedly confusing 

21 in that the Claim can be interpreted to see the Classifier performing the entire 

22 recommended test from the Explicit System, in actuality, the test is performed by the 

23 Explicit System on the Classifier, and part of the test "performed" by the Classifier is 
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1 only the output of the objective weighted score and essentially the profit from the test 

2 performed. 
3 

4 In conclusion, the use of the word "on" in Claim 10 is consistent with the 

5 specification and the overall model of the present invention, in that the Explicit System 

6 performs the recommended test on the Classifier in order to provide the Classifier with a 

7 profit and cost estimate of performing the actual test in the real world. While the 

8 Classifier "performs" the function of assembling the profit from the test performed, the 

9 overall test is performed by the Explicit System "on" the Classifier to eventually 

1 0 determine which test maximizes the profit and thus which test to then recommend to the 

1 1 Implicit System. Thus, the use of the word "on" is appropriate. 
12 

13 Claim 10 therefore satisfies the requirements of 35 USC 1 12, second paragraph. 

14 

15 

16 

17 

18 

19 

20 

21 

22 

23 
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CONCLUSION 

For the extensive reasons advanced above, the Appellant respectfully contends 
that each claim is patentable. Therefore, reversal of all rejections and objections is 
courteously requested at this time. 

Please charge any shortage of fees due in connection with the filing of this paper, 
to deposit account no. 50-2691 and please credit any excess fees to such deposit account. 
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Claims 



What is claimed is: 

1 . An apparatus for incorporating decision making into classifiers to provide efficient 
test recommendations, the apparatus comprising: 



a. an explicit system and a classifier each configured to receive a system state 
dataset the explicit system connected with the classifier and operative to 
iteratively perform a combinatory search procedure based on the system state 
dataset to develop a next test recommendation for the classifier, whereby the 
classifier performs the next test to generate an objective weighted score; 

b. a profit module connected with the classifier and with the explicit system 
to receive the objective weighted score from the classifier, to add subjective 
value to the objective weighted score to determine a profit for the test, and to 
provide the profit to the explicit system to enable the explicit system to i. 
assess the value of its next test recommendation and ii. to iteratively generate 
a best test recommendation based on the maximization of the profit; 

c. an implicit system configured to receive a system state dataset, and 
connected with the explicit system to receive the best test recommendation for 
each system state dataset, and to act as a function estimator to learn to 
associate best test recommendations with the system state dataset in order to 
mimic the explicit system, thereby to enable rapid decision making in 
situations that are either urgent or well-known. 



2. An apparatus for incorporating decision making into classifiers to provide efficient 
test recommendations as set forth in claim 1, wherein the explicit system and the 
implicit system are configured to provide test recommendations to a controller. 
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1 3. An apparatus for incorporating decision making into classifiers to provide efficient 

2 test recommendations as set forth in claim 1, wherein the implicit system is a neural 

3 network. 
4 

5 4. An apparatus for incorporating decision making into classifiers to provide efficient 

6 test recommendations at set forth in claim 3, wherein the neural network is a radial 

7 basis neural network. 
* 8 

-9 5. An apparatus for incorporating decision making into classifiers to provide efficient 

10 test recommendations as set forth in claim 3, wherein the combinatory search 

1 1 procedure performed by the explicit system is simulated annealing. 

12 

13 6. An apparatus for incorporating decision making into classifiers to provide efficient 

14 test recommendations as set forth in claim 5, wherein the explicit system and the 

1 5 profit module may be separated from the apparatus after the implicit system 

16 sufficiently mimics the explicit system. 
17 

18 7. An apparatus for incorporating decision making into classifiers to provide efficient 

19 test recommendations as set forth in claim 1, wherein the system state is a vector. 
20 

21 8. An apparatus for incorporating decision making into classifiers to provide efficient 

22 test recommendations as set forth in claim 1, wherein the classifier is a probabilistic 

23 model. 
24 

25 9. An apparatus for incorporating decision making into classifiers to provide efficient 

- 26 test recommendations as set forth in claim 8, wherein the classifier is Bayesian. 
27 

28 10. A computerized method for enhancing decision making in a classifier system, 

29 wherein the classifier system includes an explicit system and a classifier, each 
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1 configured to receive a system state dataset, with the explicit system connected with 

2 the classifier; a profit module connected with the classifier and with the explicit 

3 system; and an implicit system configured to receive a system state dataset, and 

4 connected with the explicit system, the computerized method comprising the steps of: 

5 a. receiving a system state dataset in the explicit system, the classifier, and the 

6 implicit system; 

7 b. determining in the explicit system, based on the feature set, a recommended 

8 test; 

9 c. performing the recommended test on the classifier; 

1 0 d. determining, via the profit module, the profit from the test performed on the 

1 1 classifier; 

12 e. detecting whether the test performed on the classifier maximizes the profit; 

13 f. performing the receiving step a through the detecting step e until a test is found 

14 which maximizes the profit; 

1 5 g. training the implicit system with the system state dataset and the test which 

16 maximizes the profit; and 

17 h. repeating steps a through g until a desired level of training of the implicit 

1 8 system is reached. 
19 

20 1 1 . A computerized method for enhancing decision making in a classifier system as set 

21 forth in claim 10, wherein the test that maximizes the profit is provided by either the 

22 explicit system or the implicit system to a controller. 

23 

24 12. A computerized method for enhancing decision making in a classifier system as set 

25 forth in claim 10, wherein the implicit system used is a neural network. 
26 

27 13. A computerized method for enhancing decision making in a classifier system as set 

28 forth in claim 12, wherein the implicit system used is a radial basis neural network. 
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1 

2 14. A computerized method for enhancing decision making in a classifier system as set 

3 forth in claim 12, wherein the determining step b is performed by the explicit system 

4 using a combinatory search procedure. 

5 

6 1 5. A computerized method for enhancing decision making in a classifier system as set 

7 forth in claim 14, wherein the combinatory search procedure performed by the 

8 explicit system in the determining step b is simulated annealing. 

9 

10 16. A computerized method for enhancing decision making in a classifier system as set 

1 1 forth in claim 15, wherein the explicit system and the profit module used may be 

12 separated from the classifier system after the implicit system sufficiently mimics the 

13 explicit system. 
14 

15 1 7. A computerized method for enhancing decision making into classifiers to provide 

16 efficient test recommendations as set forth in claim 10, wherein the system state is a 

17 vector. 
18 

19 1 8. A computerized method for enhancing decision making into classifiers to provide 

20 efficient test recommendations as set forth in claim 10, wherein the classifier is a 

2 1 probabilistic model. 
22 

23 19. A computerized method for enhancing decision making into classifiers to provide 

24 efficient test recommendations as set forth in claim 18, wherein the classifier is 

25 Bayesian. 
26 

27 20. A computerized method for enhancing decision making in classifiers to provide 

28 efficient test recommendations, the computerized method comprising the steps of: 
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1 a. providing an explicit system and a classifier each configured to receive a 

2 system state dataset, with the explicit system connected with the classifier, 

3 and operative to iteratively perform a combinatory search procedure based on 

4 the system state dataset to develop a next test recommendation for the 

5 classifier, whereby the classifier performs the next test to generate an 

6 objective weighted score; 

7 b. providing a profit module connected with the classifier and with the 

8 explicit system to receive the objective weighted score from the classifier, to 

9 add subjective value to the objective weighted score to determine a profit for 

1 0 the test, and to provide the profit to the explicit system to enable the explicit 

1 1 system to assess the value of its next test recommendation, and, iteratively, to 

12 generate a best test recommendation based on the maximization of the profit; 

13 c. providing an implicit system configured to receive a system state dataset, and 

14 connected with the explicit system to receive the best test recommendation for 

1 5 each system state dataset, and to act as a function estimator to learn to 

16 associate best test recommendations with the system state dataset in order to 

1 7 mimic the explicit system, thereby to enable rapid decision making in 

1 8 situations that are either urgent or well-known. 
19 

20 21 . A computerized method for enhancing decision making in a classifier system as set 

21 forth in claim 20, wherein the explicit system and the implicit system are further 

22 configured to provide the test recommendation to a controller. 

23 

24 22. A computerized method for enhancing decision making in a classifier system as set 

25 forth in claim 20, wherein the implicit system provided is a neural network. 
26 

27 23. A computerized method for enhancing decision making in a classifier system as set 

28 forth in claim 22, wherein the implicit system provided is a radial bias neural 

29 network. 
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1 

2 24. A computerized method for enhancing decision making in a classifier system as set 

3 forth in claim 22, wherein the explicit system provided performs the combinatory 

4 search procedure by use of simulated annealing. 

5 

6 25. A computerized method for enhancing decision making in a classifier system as set 
. 7 forth in claim 24, wherein the explicit system provided and the profit module 

8 provided may be separated from the classifier system provided after the implicit 

9 system sufficiently mimics the explicit system. 
10 

11 26. A computerized method for enhancing decision making in classifiers to provide 

12 efficient test recommendations as set forth in claim 20, wherein the system state is a 

13 vector. 
14 

15 27. A computerized method for enhancing decision making in classifiers to provide 

16 efficient test recommendations as set forth in claim 20, wherein the classifier is a 

17 probabilistic model. 
18 

19 28. A computerized method for enhancing decision making in classifiers to provide 

20 efficient test recommendations as set forth in claim 27, wherein the classifier is 

21 Bayesian. 
22 
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APPENDIX B 
Declaration of Inventor Bradley C Love 



I, Bradley C. Love, of Austin, Texas, am the inventor for application 09/686,1 12, 
the "Method and Apparatus for Incorporating Decision Making into Classifiers." I am an 
assistant professor at the University of Texas at Austin in the Center for Computation and 
Cognitive Processes, with a Ph,D* in Cognitive Psychology and a B,S. in Cognitive and 
Linguistic Sciences. I have worked in the area of computer science and cognitive 
processes for over 10 years, with numerous grants and publications in the field. I do 
hereby declare the following: 

A Vector is not the most natural way to represent the subject matter of the present 
application. The system state of the present application is not confined to values that are 
known and unknown. Many cases require searching for the next test by navigating a 
matrix structure for situations in which there are test dependencies. The state of the 
search process is thus pan of the system state. Additionally, the complete state of the 
model could encompass the above, plus the specification of the classifier, the implicit 
system, and explicit system. 

With regard to the use of the word "mimic," the notion of a learning model 
coming to approximate, estimate, or mimic some underlying function is accepted in 
machine learning and statistics. For example, gradient descent models, such as error- 
minimizing neural networks, come to approximate or mimic an underlying function over 



Page 1 of 2 



US App,No, 09/686,112 Appeal Brief Appendix B 

1 the course of learning. In the present application, the function being approximated is the 

2 one defined by the input/output mapping of the explicit system. What constitutes a 

3 sufficient approximation or adequate mimicry cannot be defined outside of the context of 

4 a specific application and evaluation metric. Each application will have different costs 

5 associated with gathering training data, making timely predictions, and making errors, 

6 All of these factors (and more) affect what is adequate mimicry. Matching the outputs of 

7 a system 60% of the time may be adequate in some situations, whereas matching 99.9% 

8 may be inadequate for other applications. In cases where time is of the essence and the 

9 explicit system cannot respond fast enough, any prediction may be desirable. Learning 

10 theoreticians working in the PAC (Probably Approximately Correct) learning framework 

1 1 formally specify the probability that an estimation will fall outside certain error bounds 

12 given a certain number of training examples for certain classes of problems, but these 

1 3 analyses are not applicable to many practical situations, nor do they dictate what the 

14 bounds should be (as discussed above). 
15 

16 

17 Cff^t/oy 

. 18 Bradley C. Love, Inventor Date 
19 
20 
21 
22 
23 
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